STFE-Net: A Spatial-Temporal Feature Extraction Network for Continuous Sign Language Translation

The main challenge of continuous sign language translation (CSLT) lies in the extraction both discriminative spatial features and temporal features. In this paper, a spatial-temporal feature network (STFE-Net) is proposed for CSLT, which optimally fuses features, extracted by (SFE-Net) (TFE-Net), respectively. SFE-Net performs pose estimation presenters sign-language videos. Based on COCO-WholeBody, 133 key points are abbreviated to 53 points, according characteristics language. High-resolution performed hands, along with whole-body estimation, obtain finer-grained hand words then fed TFE-Net, based Transformer relative position encoding. dataset Chinese was created used evaluation. STFE-Net achieves Bilingual Evaluation Understudy (BLEU-1, BLEU-2, BLEU-3, BLEU-4) scores 77.59, 75.62, 74.25, 72.14, Furthermore, our also evaluated two public datasets, RWTH-Phoenix-Weather 2014T CLS. BLEU-1, BLEU-3 BLEU-4 achieved method former 48.22, 33.59, 26.41 22.45, respectively, corresponding 61.54, 58.76, 57.93 57.52, latter dataset. Experiment results show that model can achieve promising performance. If any reader needs code or dataset, please email [email protected].